-
Notifications
You must be signed in to change notification settings - Fork 418
[Custom Transactions] Add TxBuilder::get_next_commitment_stats
#3921
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
👋 Thanks for assigning @carlaKC as a reviewer! |
We discussed earlier passing the entire list of HTLCs to So we prefer to add a single method that surfaces this limit to channel, and then let channel sort HTLCs depending on whether their value sits above or below that amount. As a result, builders of custom transactions will only have a say on where the dust limit sits, and won't be able to choose arbitrary subsets for dust and non-dust HTLCs. |
Is the main motivation for this to get all of the commitment-related logic out of
The key questions we're looking to answer seems to be "can I afford a commitment with this theoretical dust/nondust htlc"? This does seem to be something we could move into Sadly here's no getting around needing to know the dust limit if we want to clamp our capacity to that value, so we'd still need to surface
Tempting to suggest just adding an so tl;dr: I'd be interested in seeing what trying to pull more of the dust logic out into Meta note: This has got a lot of overlap with 3bb0586, so I think we should either:
|
I would say we focus on the latter for this PR, and let the former be the overarching goal :)
Let me know what you think of this new direction here. Still have some clunkiness to resolve, but that's what it's looking like right now.
|
Definitely let me rebase on top of your PR, you've rebased once already :) |
12fd6c3
to
067bfa0
Compare
🔔 1st Reminder Hey @carlaKC! This PR has been waiting for your review. |
I like this approach! Definitely prefer being able to move a lot of the dust / second stage tx reasoning into the builder 👍 |
🔔 2nd Reminder Hey @carlaKC! This PR has been waiting for your review. |
I'm aware I still owe a follow-up PR from the previous PR in this project, will push that soon in a separate PR :) |
lightning/src/sign/tx_builder.rs
Outdated
fn on_holder_tx_dust_exposure_msat( | ||
&self, dust_buffer_feerate: u32, holder_dust_limit_satoshis: u64, | ||
channel_type: &ChannelTypeFeatures, htlcs: &[HTLCAmountDirection], | ||
) -> u64; | ||
fn on_counterparty_tx_dust_exposure_msat( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not really sure I understand the point of these two methods if we have htlc_success_timeout_dust_limit
as a method - if the API includes a method to say "hey, for an HTLC of type X what is the threshold where its dust", why bother with a method to say "given these HTLCs, how many are dust?", it seems the second can be calculated from the first, no? The same applies for passing the HTLC list to commit_tx_fee_sat
.
Alternatively, we could drop the htlc_success_timeout_dust_limits
method (if we want to not require there be some strict threshold to indicate when an HTLC is dust, though I think its a fine assumption to bake into the API, I dunno why someone would want to have some HTLCs be dust and others at the same value not be) and have a more generic "is HTLC dust on the next counterparty/local commitment transaction" call, but even there it seems like we don't need all of these methods.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sounds good yes I initially had a version of the API that matches what you describe. We wanted to see what it would look like if we moved more logic out of channel into TxBuilder
, but I agree we only really need the dust limits.
I will clean up those methods, and add TxBuilder
methods for 1) the dust limit 2) the htlc endogenous fees (to calculate counterparty dust exposure).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I mean don't get me wrong, I'd love to move more logic out of channel.rs
into TxBuilder
, that would be great, but we also don't want to have an overlapping API where we have two ways to do things over the API. Also, we want to eventually make TxBuilder
public, so it would be nice to avoid adding too much complexity (in the form of a ton of different methods).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
FWIW the real blocker on the more ambitious / complex API was the get_available_balances
method - at the moment this method requires an exact dust limit above which all HTLCs are non-dust.
I have some code written that moves most of get_available_balances
behind TxBuilder
, but the complexity is not worth it, especially if we can make the assumption you describe above about the strict dust-vs-nondust threshold.
👋 The first review has been submitted! Do you think this PR is ready for a second reviewer? If so, click here to assign a second reviewer. |
TxBuilder
set the HTLC dust limit
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #3921 +/- ##
==========================================
- Coverage 88.93% 88.85% -0.09%
==========================================
Files 174 175 +1
Lines 123876 127872 +3996
Branches 123876 127872 +3996
==========================================
+ Hits 110173 113623 +3450
- Misses 11250 11686 +436
- Partials 2453 2563 +110
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
for what it can be worth it I have reviewed 1eb1c9e and it looks correct to me.
As for my understanding on the reasoning for the change, motivation seems to be to move usage of second_stage_tx_fees_sat
and htlc_tx_fees_sat
to TxBuilder
in order to abstract away usage of weight of htlc transactions away from channel.rs
.
Rebase: diff
|
0904e98
to
3b851ba
Compare
Rebase: full diff
|
@carlaKC I believe the PR is ready for a deeper look :) Can't quite figure out why CI fails to build, it does pass on my machine. I thought we could focus on the first two commits for now, and we push the validation / prediction commits to the next PR in this series. Let me know what you'd prefer. I've pushed the validation / prediction commits here too so you have the context. Also see their commit messages for further description of the changes. |
@carlaKC A correction: In the b16779a commit message, I say "This commit also includes outbound HTLC additions in the holding cell in the fees on the next remote commitment transaction; they were previously not included." This is wrong, they are already included via the additional HTLC parameter of |
🔔 1st Reminder Hey @carlaKC! This PR has been waiting for your review. |
I also plan to expand the commit messages to describe in detail how we now read the feerates we use to calculate the next commitment stats. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewed first two commits - API looks good!
I don't have strong feelings on how to split this up - happy to separate API/usage or do both in one go. @tankyleo lmk your preference and I'll review accordingly.
I'm ready for this to get a rebase/squash so it can pick up some minor relevant V3 channel changes.
/// We take the conservative approach and only assume that a HTLC will | ||
/// not be in the next commitment when it is guaranteed that it won't be. | ||
#[allow(dead_code)] | ||
#[rustfmt::skip] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: don't skip formatting on new code?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
On my machine rustfmt is not consistent when formatting get_next_commitment_htlcs
and get_next_commitment_value_to_self_msat
, and also makes a big mess of get_next_commitment_value_to_self_msat
:)
Given a snapshot of the lightning state machine, `TxBuilder::get_next_commitment_stats` calculates the transaction fees, the dust exposure, and the holder and counterparty balances (the balances themselves do *not* account for the transaction fee).
In upcoming commits, these methods will serve as proxies to `SpecTxBuilder::get_next_commitment_stats` in all validation of channel updates in `ChannelContext`. Eventually, these methods will completely replace `get_pending_htlc_stats`, and `get_next_{local, remote}_commit_tx_fee_msat`. When predicting the HTLCs on next commitment, we take the conservative approach and only assume that a HTLC will not be in the next commitment when it is guaranteed that it won't be.
`ChannelContext::get_pending_htlc_stats` predicts that the set of HTLCs on the next commitment will be all the HTLCs in `ChannelContext.pending_inbound_htlcs`, and `ChannelContext.pending_outbound_htlcs`, as well as all the outbound HTLC adds in the holding cell. This is an overestimate: * Outbound HTLC removals which have been ACK'ed by the counterparty will certainly not be present in any *next* commitment, even though they remain in `pending_outbound_htlcs`. * Outbound HTLCs in the `RemoteRemoved` state, will not be present in the next *local* commitment. * Outbound HTLCs in the `LocalAnnounced` state have no guarantee that they were received by the counterparty before she sent the `update_fee`. * Outbound `update_add_htlc`'s in the holding cell are certainly not known by the counterparty, and we will reevaluate their addition to the channel when freeing the holding cell. * Inbound HTLCs in the `LocalRemoved` state will not be present in the next *remote* commitment. `ChannelContext::next_local_commit_tx_fee_msat` over-counts outbound HTLCs in the `LocalAnnounced` and `RemoteRemoved` states, as well as outbound `update_add_htlc`'s in the holding cell. `ChannelContext::next_remote_commit_tx_fee_msat` over-counts inbound HTLCs in the `LocalRemoved` state, as well as outbound HTLCs in the `LocalAnnounced` state. This commit stops using these functions in favor of the newly added `ChannelContext::get_next_{local, remote}_commitment_stats` methods, and fixes the issues described above. If we are the funder, we also check that adding this inbound HTLC doesn't increase the commitment transaction fee to the point of exhausting our balance on the local commitment. Previously, we would only subtract the anchors from `funding.value_to_self_msat`; we now also subtract the outbound HTLCs on the next local commitment from `funding.value_to_self_msat` before checking if we can afford the additional transaction fees. Inbound `LocalRemoved` HTLCs that were **not** successful are now credited to `remote_balance_before_fee_msat` as they will certainly not be on the next remote commitment. We previously debited these from the remote balance to arrive at `remote_balance_before_fee_msat`. When calculating dust exposure, we now take a buffer from the currently committed feerate, and ignore any fee updates in `ChannelContext.pending_update_fee`.
`ChannelContext::get_pending_htlc_stats` predicts that the set of HTLCs on the next commitment will be all the HTLCs in `ChannelContext.pending_inbound_htlcs`, and `ChannelContext.pending_outbound_htlcs`, as well as all the outbound HTLC adds in the holding cell. This is an overestimate: * Outbound HTLC removals which have been ACK'ed by the counterparty will certainly not be present in any *next* commitment, even though they remain in `pending_outbound_htlcs`. * Outbound HTLCs in the `RemoteRemoved` state, will not be present in the next *local* commitment. * Outbound HTLCs in the `LocalAnnounced` state have no guarantee that they were yet received by the counterparty. * Outbound `update_add_htlc`'s in the holding cell are certainly not known by the counterparty, and we will reevaluate their addition to the channel when freeing the holding cell. * Inbound HTLCs in the `LocalRemoved` state will not be present in the next *remote* commitment. This commit stops using `get_pending_htlc_stats` in favor of the newly added `ChannelContext::get_next_{local, remote}_commitment_stats` methods, and fixes the issues described above. `ChannelContext::next_remote_commit_tx_fee_msat` counts inbound HTLCs in the `LocalRemoved` state, as well as outbound HTLCs in the `LocalAnnounced` state. We now do not count them for the same reasons described above. Inbound `LocalRemoved` HTLCs that were **not** successful are now credited to `remote_balance_before_fee_msat` as they will certainly not be on the next remote commitment. We previously debited these from the remote balance to arrive at `remote_balance_before_fee_msat`. We now always check holder dust exposure, whereas we previously would only do it if the incoming HTLC was dust on our own commitment transaction. Furthermore, dust exposure calculations now take a buffer from the currently committed feerate, and ignore any fee updates in `ChannelContext.pending_update_fee`.
`ChannelContext::get_pending_htlc_stats` predicts that the set of HTLCs on the next commitment will be all the HTLCs in `ChannelContext.pending_inbound_htlcs`, and `ChannelContext.pending_outbound_htlcs`, as well as all the outbound HTLC adds in the holding cell. This is an overestimate: * Outbound HTLC removals which have been ACK'ed by the counterparty will certainly not be present in any *next* commitment, even though they remain in `pending_outbound_htlcs`. * Outbound HTLCs in the `RemoteRemoved` state, will not be present in the next *local* commitment. * Outbound HTLCs in the `LocalAnnounced` state have no guarantee that they were received by the counterparty before she sent the `update_fee`. * Outbound `update_add_htlc`'s in the holding cell are certainly not known by the counterparty, and we will reevaluate their addition to the channel when freeing the holding cell. * Inbound HTLCs in the `LocalRemoved` state will not be present in the next *remote* commitment. This commit stops using `get_pending_htlc_stats` in favor of the newly added `ChannelContext::get_next_{local, remote}_commitment_stats` methods, and fixes the issues described above. We now always calculate dust exposure using a buffer from `msg.feerate_per_kw`, and not from `max(self.feerate_per_kw, msg.feerate_per_kw)`.
`ChannelContext::get_pending_htlc_stats` predicts that the set of HTLCs on the next commitment will be all the HTLCs in `ChannelContext.pending_inbound_htlcs`, and `ChannelContext.pending_outbound_htlcs`, as well as all the outbound HTLC adds in the holding cell. This is an overestimate: * Outbound HTLC removals which have been ACK'ed by the counterparty will certainly not be present in any *next* commitment, even though they remain in `pending_outbound_htlcs` (I refer to states `AwaitingRemoteRevokeToRemove` and `AwaitingRemovedRemoteRevoke`). * Outbound HTLCs in the `RemoteRemoved` state, will not be present in the next *local* commitment. * Inbound HTLCs in the `LocalRemoved` state will not be present in the next *remote* commitment. `ChannelContext::build_commitment_stats(funding, true, true, ..)` makes these errors when predicting the HTLC count on the remote commitment: * Inbound HTLCs in the state `RemoteAnnounced` are not included, but they will be in the next remote commitment transaction if the local ACK's the addition before producing the next remote commitment. * Inbound HTLCs in the state `AwaitingRemoteRevokeToAnnounce` are not included, even though the local has ACK'ed the addition. * Outbound HTLCs in the state `AwaitingRemoteRevokeToRemove` are counted, even though the local party has ACK'ed the removal. This commit replaces these functions in favor of the newly added `ChannelContext::get_next_{local, remote}_commitment_stats` methods, and fixes the issues described above. We now always calculate dust exposure using a buffer from `msg.feerate_per_kw`, and not from `max(feerate_per_kw, self.feerate_per_kw, self.pending_update_fee)`.
Anytime we build a (feerate, nondust-htlc-count, fee) pair, cache it, and check that the fee matches if the feerate and nondust-htlc-count match when building a commitment transaction.
The cached fee is never checked in the current test suite.
Rebase: squash, nits, edit commit messages; full diff |
🔔 1st Reminder Hey @carlaKC! This PR has been waiting for your review. |
let include_counterparty_unknown_htlcs = false; | ||
// Don't include the extra fee spike buffer HTLC in calculations | ||
let fee_spike_buffer_htlc = 0; | ||
let next_remote_commitment_stats = self.get_next_remote_commitment_stats(funding, Some(HTLCAmountDirection { outbound: false, amount_msat: msg.amount_msat }), include_counterparty_unknown_htlcs, fee_spike_buffer_htlc, self.feerate_per_kw, dust_exposure_limiting_feerate); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we hit the following here?
- Remote peer sends an
update_add_htlc
for an amount > their current balance get_next_remote_commitment_stats
includes the candidate innext_commitment_htlcs
(get_next_commitment_htlcs
always adds the candidate)get_next_commitment_stats
includes the large htlc ininbound_htlcs_value_msat
and thechecked_sub
onvalue_to_counterparty_msat
'sexpect
fails
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
woah thank you ! yes will be sure to fix this :)
Do not panic if the counterparty adds a HTLC with a value greater than its remaining balance!
For consistency, include the number of additional nondust HTLCs in the total nondust HTLC count; commit_tx_fee_sat also includes these.
🔔 1st Reminder Hey @carlaKC! This PR has been waiting for your review. |
and